Program Evaluation's Path to Greater Policy Relevance: Learning From Rossi's Iron Laws.

In a 1987 article, Peter R. Rossi promulgated "The Iron Law of Evaluation and Other Metallic Rules." The Metallic Laws were meant as an informal (and humorous) overstatement of the weakness of contemporary evaluations of social programs. Rossi' s underlying worry was not so much about the state of evaluation technology in the abstract, but, rather, in its inability to advance our broad understanding of social problems and what to do about them---in other words, to make evaluation policy relevant. Rossi attributed the continuing failure to develop successful "large-scale social programs" to the failure to build a strong knowledge base for this kind of "social engineering." The qualities of studies that enable such accumulated learning are variously labeled "external validity," "generalizability," "applicability," or "transferability." This Special Issue includes five papers that seek to explore and apply this understanding.

What Makes a Good Match? Predictors of Quality Mentorship Among Doctoral Students.

Quality mentoring promotes graduate student success. Despite an abundance of practical advice, empirical evidence regarding how to match mentees and mentors to form quality mentoring relationships is lacking. Here, we examine the influence of variables theorized to predict mentorship support and quality in a national sample of 565 science doctoral students from 70 universities in 38 states. Our structural equation modeling results indicate that mentor rank, mentee capital, and the relationship matching mechanism (direct admissions, rotations) were not associated with higher-quality relationships. We found no support for the widely held belief that students whose mentors shared their gender, race, or ethnicity experienced greater mentorship quality. Rather, mentees who shared attitudes, beliefs, and values with their mentor, or whose mentors displayed greater cultural awareness experienced more supportive, higher quality mentoring. Furthermore, these patterns were largely consistent across both mentee and mentor demographic groups. These results highlight the potential benefits of pairing mentees and mentors who share personal and intrinsic qualities rather than demographic or surface-level attributes. Our findings also indicate that graduate students from marginalized backgrounds can be effectively mentored by faculty who are demographically dissimilar if their mentors engage in culturally aware mentorship.

An integrative review of parent education approaches in sport: Considerations for program planning and evaluation.

In recent years, there has been an increase in the delivery and evaluation of parent education programs within youth sport. Subsequently, some recent reviews of these programs have been conducted. However, one consistent issue across many of the programs and associated review papers is the lack of an appropriate evaluation framework to guide the planning or associated reporting of the outcomes of the interventions. This has limited understanding of the overall impact of sport parenting interventions. Thus, the purposes of the current study were as follows: (a) to identify commonalities in the reporting and evaluation of parent education programs; (b) to identify gaps in the reporting and evaluation of parent education programs; (c) to draw these insights together to provide suggestions regarding how the RE-AIM could be used to enhance planning and evaluation of evidence-based programs for parent education in sport. Specifically, utilizing the RE-AIM framework to provide insights into pertinent evaluation metrics, this integrative review aimed to identify commonalities and gaps in the reporting of parent education programs. The RE-AIM framework considers the essential elements to assess the external and internal validity of interventions through five dimensions: Reach, Effectiveness, Adoption, Implementation, and Maintenance (Am J Public Health. 1999;89(9):1322-1327). Subsequently, the review aimed to provide suggestions regarding strategies to enhance the planning and evaluation of evidence-based programs for parent education in sport. Overall, the analysis demonstrated that most studies presented some pertinent evaluation information related to the RE-AIM framework, such as the number of participants and contacts made, the measures used, and the program level. However, the studies also lacked information on participant exclusion criteria, the method used to select the delivery agent (e.g., parents engaged in the program), and cost measures. Overall, the current study identified various areas where programs could be enhanced, specifically related to reporting procedural elements (e.g., program design, target population, and costs) pertaining to the implementation of parent education programs.

Effect of a Legal Clinic Program Within an Urban Primary Health Care Center on Social Determinants of Health: A Program Evaluation.

BACKGROUND: Individuals living in poverty often visit primary healthcare clinics for health problems stemming from unmet legal needs. We examined the impact of a medical-legal partnership on improving the social determinants of health (SDoH), health-related quality of life, and perceived health status of attendees of a Legal Clinic Program (LCP). METHODS: This was a pre-post program evaluation of a weekly LCP established within an urban primary healthcare clinic to provide free legal consultation. Patients aged 18 years or older were either approached or referred to complete a screening tool to identify potential legal needs. Those identified with potential legal needs were offered an appointment with LCP lawyers who provided legal counsel, referrals, and services. For those who attended the LCP, changes in SDoH and health indicators were collected via a self-reported survey 6 months after they attended the LCP and compared to their baseline scores using paired t-tests, McNemar's test for paired proportions, and the Wilcoxon Signed Rank Test for related samples. RESULTS: During the 6-month evaluation period, 31 participants attended the LCP and completed both the baseline and 6-month surveys; 67.8% were female, 64.5% were white, 90.3% were not working full-time, and 61.3% had a household income of $700 to 1800 per month. At follow-up, 25.8% were receiving at least 1 new benefit and there was a statistically significant reduction in food insecurity (35.5% vs 9.7%, P < .05). Also, perceived health status using the visual analog scale (ranges from 0 to 100) significantly improved from 42.5 points (SD = 25.3) at baseline to 56.6 points (SD = 19.6) after 6 months (P < .05). CONCLUSIONS: The LCP has the potential to improve the health and wellbeing of patients in primary healthcare clinics by addressing unmet legal needs and SDoH.

Statistical estimation of fatal and serious injuries saved by iRAP protocols in 74 countries.

OBJECTIVE: Road traffic crashes cause 1.19 million deaths and millions more injuries annually. The persistently high burden has drawn attention from national and international stakeholders worldwide. Unsafe road infrastructure is one of the major risk factors for traffic safety, particularly in low- and middle-income countries. METHODS: Aiming to eliminate high-risk roads in all countries, the International Road Assessment Programme (iRAP) developed a robust and evidence-based approach to support country transportation agencies. RESULTS: Thus far, the iRAP protocols have been used to collect 1.8 million kilometers of Crash Risk Mapping and 1.5 million kilometers of Star Rating and FSI estimations in 128 countries. Deploying an observational before-and-after (or pre-post) study design, this report estimated the fatal and series injuries (FSI) saved through use of the iRAP protocols. The study is based on 441,753 kilometers of assessed roads from 1,039 projects in 74 countries. Our results show that the implementation of iRAP's proposed countermeasures saves about 159,936 FSI annually. Throughout the lifetime of the implemented countermeasures, a total of 3.2 million FSI could be saved. CONCLUSION: While quantifying the success of the iRAP protocols, our results suggest an opportunity to save many millions more lives on the roads through expanding iRAP implementation to more regions and countries.

Program evaluation of a school-based mental health and wellness curriculum featuring yoga and mindfulness.

BACKGROUND: Interest in the effectiveness of mindfulness-based interventions such as yoga in primary schools has grown. Evidence shows promise, as youth who engage in yoga to promote mindfulness show improved coping skills, increased socio-emotional competence and prosocial skills, academic performance, attention span, and ability to deal with stress. OBJECTIVE: This study reports the results of a program evaluation of a universal health and wellness curriculum, Pure Power, designed to teach youth yoga techniques, mindfulness, and emotion regulation. METHODS: A non-randomized comparison design examined outcomes among participants from schools that completed the intervention with highest fidelity of implementation (n = 461) and from students in matched comparison schools (n = 420). Standard measures of coping, emotion regulation and emotion dysregulation, spelling, and math achievement were collected. RESULTS: Analyses suggest the youth in the intervention schools demonstrated relative improvement on measures of emotion regulation, spelling, and math. CONCLUSIONS: Challenges in implementation in real-life settings are vital to identify. The data provide some real-world evidence for the effectiveness of a universal health and wellness curriculum on emotion regulation and positive academic outcomes. Training school staff to deliver the intervention may foster implementation. Future research should test the effectiveness of who delivers the intervention; for example, teacher-delivered groups vs. other wellness personnel.

Development, implementation, and evaluation of an undergraduate family medicine program in the United Arab Emirates.

BACKGROUND: Healthcare systems rely on well-trained family medicine physicians who can offer continuous quality services to their communities and beyond. The American Academy of Family Physicians and the World Organization of Family Doctors recommend that medical curricula should have adequately supervised education and training of the learners in family medicine during their preclinical and clinical placements. However, some medical schools don't have a comprehensive family medicine program to prepare graduates who can meet the community needs. This work aims to report the essential steps for the development, implementation, and evaluation of the family medicine program at the College of Medicine at the University of Sharjah in United Arab Emirates. METHODS: We used the Kern's 6-step model to describe the development, implementation, and evaluation of the family medicine program. This includes problem identification, needs assessment, goals setting, educational strategies, implementation, and evaluation. During 2014-2022, we longitudinally collected essential information about the family medicine program from different stakeholders including the feedback of clinical coordinators, adjunct clinical faculty, and medical students at the end-of-clerkship. All responses were analysed to determine the effective implementation and evaluation of the family medicine program. RESULTS: Over the course of 8 academic years, 804 medical students, 49 adjunct clinical faculty and three College of Medicine faculty participated in the evaluation of the family medicine program. The majority of respondents were satisfied with various aspects of the family medicine program, including the skills gained, the organisation of program, and the variety of clinical encounters. The medical students and adjunct clinical faculty suggested the inclusion of e-clinics, faculty development program, and the expansion of more clinical sites for the effectiveness of the family medicine program. CONCLUSIONS: We report a successful development, implementation, and evaluation of the family medicine program in United Arab Emirates with a positive and impactful learning experience. More attention should be paid towards a suitable representation of family medicine program in the medical curriculum with focused and targeted educational plans for medical students.

Direct Admission Program Implementation: A Qualitative Analysis of Variation Across Health Systems.

OBJECTIVES: Direct admission (DA) to the hospital has the potential to improve family satisfaction and timeliness of care by bypassing the emergency department. Using the RE-AIM implementation framework, we sought to characterize variation across health systems in the reach, effectiveness, adoption, and implementation of a DA program from the perspectives of parents and multidisciplinary clinicians. METHODS: As part of a stepped-wedge cluster randomized trial to compare the effectiveness of DA to admission through the emergency department, we evaluated DA rates across 69 clinics and 3 health systems and conducted semi-structured interviews with parents and clinicians. We used thematic analysis to identify themes related to the reach, effectiveness, adoption, and implementation of the DA program and applied axial coding to characterize thematic differences across sites. RESULTS: Of 2599 hospitalizations, 171 (6.6%) occurred via DA, with DA rates varying 10-fold across health systems from 0.9% to 9.3%. Through the analysis of 137 interviews, including 84 with clinicians and 53 with parents, we identified similarities across health systems in themes related to perceived program effectiveness and patient and family engagement. Thematic differences across sites in the domains of program implementation and clinician adoption included variation in transfer center efficiency, trust between referring and accepting clinicians, and the culture of change within the health system. CONCLUSIONS: The DA program was adopted variably, highlighting unique challenges and opportunities for implementation in different hospital systems. These findings can inform future quality improvement efforts to improve transitions to the hospital.

Supporting underrepresented students in health sciences: a fuzzy cognitive mapping approach to program evaluation.

BACKGROUND: The Students Training in Academia, Health, and Research (STAHR) Program at the University of Missouri-Kansas City (UMKC) strives to help students from low-income families that have experienced educational challenges due to poverty and prepare them to enter, persist, and graduate from a health sciences degree program at UMKC. Students in the program participated in fuzzy cognitive mapping (FCM) sessions to ensure that all voices of the program were heard to improve program implementation, and student success, and contribute to an equitable educational environment. METHODS: Fuzzy Cognitive Mapping sessions for the 2020-2021 cohort of students (n = 52) were conducted to assess the strengths and weaknesses in program implementation, especially through the beginning of the COVID-19 pandemic. Students' maps were coded by a team of researchers and then confirmed using confirmatory factor analysis. RESULTS: Statistical analyses reveal that mentorship, workshops, and social support helped students to work toward their goal of obtaining a professional health sciences degree, while a lack of time, remote learning, and outside stressors inhibited their opportunities for success. CONCLUSIONS: The findings from a multipronged analysis of mapping data demonstrate the value of this innovative approach to the field, especially when looking to incorporate student voices.

Evaluation of a peer-support, 'mentor mother' program in Gaza, Mozambique; a qualitative study.

BACKGROUND: Retention in prevention of mother-to-child transmission of HIV programs is critical to reduce vertical transmission. To addresses challenges with retention, Mozambique launched a peer-support program in 2018, in which HIV-positive mothers provide adherence support as mentor mothers (MMs) for HIV-positive pregnant and lactating women and HIV-exposed and infected children. METHODS: A descriptive qualitative evaluation was conducted across nine facilities in Gaza Province to assess the acceptability and barriers to implementation of the mentor mother program (MMP) among those receiving services and providing services. In-depth interviews and focus group discussions were conducted with MMs, MM supervisors, health care workers (HCWs), HIV-positive mothers enrolled in the MMP, HIV-positive mothers who declined MMP enrollment, and key informants involved in the implementation of the program. Thematic analysis identified emerging recurrent themes and patterns across the participants' responses. Data were collected between November-December 2020. RESULTS: There were initial challenges with acceptability of the MMP, especially regarding confidentiality concerns and MM roles. Sharing additional information about MMs and making small changes during the beginning of the MMP resulted in generally high acceptance of the MMP. HIV-positive mothers reported that counseling from MMs improved their understanding of the importance of anti-retroviral treatment (ART) and how to take and administer ART. HIV-positive mothers reported having reduced guilt and shame about their HIV-status, feeling less alone, and having more control over their health. MMs shared that their work made them feel valued and decreased their self-stigmatization. However, MMs also reported feeling that they had inadequate resources to perform optimal job functions; they listed inadequate transportation, insufficient stipends, and false addresses of clients among their constraints. Overall, HCWs felt that their workload was significantly reduced with MM support and wanted more MMs in the community and health facility. CONCLUSIONS: This study found that the MMP was considered a substantive and highly valued support to HIV-positive mothers, resulting in increased ART literacy among patients, improved self-reported well-being and sense of community and reduced feelings of isolation. Recommendations include strengthening MM training, increasing financial and materiel resources, additional information provided to newly enrolled mothers and support for the male partners.

Using Wash'Em to Design Handwashing Programmes for Crisis-Affected Populations in Zimbabwe: A Process Evaluation.

Wash'Em is a process that supports humanitarians in assessing and designing rapid but context-specific hygiene programmes in crises or outbreaks. The process consists of training implementers, using tools to learn from populations, and entering findings into a software which generates contextualised activities. A process evaluation of Wash'Em use was conducted in a drought-affected area in Midland province, Zimbabwe. Data were collected during the programme design and following implementation using a mix of qualitative methods. Findings were classified against the intended stages of Wash'Em, and the evaluation domains were defined by the UKRI Medical Research Council. The Wash'Em process was not fully implemented as intended. An abridged training was utilised, some of the tools for learning from populations were omitted, many of the recommended activities were not implemented, the delivery modalities were different from intended, the budget available was minimal, and the number of people exposed to activities were fewer than hoped. Despite these 'on the ground' challenges and adaptations, the Wash'Em process was considered feasible by implementers and was seen to be less top-down than most programme design approaches. The populations exposed to the intervention found the activities engaging, understood the content, and reportedly took action to improve handwashing behaviour. Programmes such as Wash'Em, which facilitate community participation and are underpinned by theory and evidence, are likely to yield positive results even if processes are followed imperfectly.

Mentoring Models to Support Transition to Practice Programs: One Size Does Not Fit All.

One-on-one mentoring is not a fit for all transition to practice programs because of the need to recruit large numbers of mentors several times a year and the cost associated with supporting many mentor/mentee relationships. A group mentoring model is sustainable because it can foster a collaborative learning environment and emphasize knowledge sharing, skill acquisition, and emotional support within the group dynamic. Models can be replicated and applied in any setting. [J Contin Educ Nurs. 2024;55(4):161-164.].

Outcomes for Adults With Intellectual and Developmental Disabilities Receiving Long-Term Services and Supports: A Systematic Review of the Literature.

The impact of long-term services and supports on the quality of life of adults with intellectual and developmental disabilities (IDD) is not well understood given the highly complex nature of researching this topic. To support future research addressing this topic, we conducted a systematic literature review of studies addressing outcomes of adults with IDD receiving long-term services and supports. Results of this review describe current outcomes for adults with IDD who receive long-term services and supports and can be used to inform program evaluation, policy development, and future research.

Protocol for an interview-based method for mapping mental models using causal-loop diagramming and realist interviewing.

Causal-loop diagramming, a method from system dynamics, is increasingly used in evaluation to describe individuals' understanding of how policies or programs do or could work ("mental models"). The use of qualitative interviews to inform model development is common, but guidance for how to design and conduct these interviews to elicit causal information in participant mental models is scant. A key strength of semi-structured qualitative interviews is that they let participants speak freely; they are not, however, designed to elicit causal information. Moreover, much of human communication about mental models-particularly larger causal structures such as feedback loops-is implicit. In qualitative research, part of the skill and art of effective interviewing and analysis involves listening for information that is expressed implicitly. Similarly, a skilled facilitator can recognize and inquire about implied causal structures, as is commonly done in group model building. To standardize and make accessible these approaches, we have formalized a protocol for designing and conducting semi-structured interviews tailored to eliciting mental models using causal-loop diagramming. We build on qualitative research methods, system dynamics, and realist interviewing. This novel, integrative method is designed to increase transparency and rigor in the use of interviews for system dynamics and has a variety of potential applications.

'First, do no harm': systematic program evaluation of an equine veterinary service-learning initiative with Indigenous communities in Canada.

BACKGROUND: Veterinary students have historically lacked meaningful experiential learning opportunities in equine medicine. At the same time, there are barriers to accessing veterinary care in Indigenous communities stemming from colonial injustices. In 2018-2019, a partnership was initiated where University of Calgary students began to provide equine veterinary services to Indigenous communities. As the first-documented equine veterinary service-learning initiative in Indigenous communities embedded in a veterinary curriculum, the purpose of the study is to systematically evaluate the program for its potential impact as part of a formative process for improvement. METHODS: Multiple parties in the program were engaged in a convergent, parallel, mixed-methods systematic program evaluation to explore the main program outcomes: (1) equine veterinary care; (2) clinical experiential student education; (3) cultural training of veterinary professionals and students; and (4) education of community members. The hypothesis was that ethical development using the "first, do no harm principle" would lead to benefits including a healthy horse population, a technically and culturally competent veterinary community, and an educated horse clientele. RESULTS: The program had a positive impact on accessibility to veterinary care and self-reported improvement in veterinary and cultural competency. In addition to the hypothesized program outcomes, additional program outcomes and effects were identified, including reciprocal learning and relationship building with the Indigenous community, leading to trust and equity-building. The students learned from both the in-community programming as well as the Indigenous community members they worked with. CONCLUSION: Program evaluation of an equine service-learning initiative in Indigenous communities reveals multiple and profound impacts including improved patient health status, wider scope of veterinary and cultural learning, strengthened relationships, and reciprocal learning with partnering Indigenous communities.

The impact of a regionally based translational cancer research collaborative in Australia using the FAIT methodology.

BACKGROUND: Translating research, achieving impact, and assessing impact are important aspirations for all research collaboratives but can prove challenging. The Hunter Cancer Research Alliance (HCRA) was funded from 2014 to 2021 to enhance capacity and productivity in cancer research in a regional centre in Australia. This study aimed to assess the impact and benefit of the HCRA to help inform future research investments of this type. METHOD: The Framework to Assess the Impact from Translational health research (FAIT) was selected as the preferred methodology. FAIT incorporates three validated methodologies for assessing impact: 1) Modified Payback; 2) Economic Analysis; and 3) Narrative overview and case studies. All three FAIT methods are underpinned by a Program Logic Model. Data were collected from HCRA and the University of Newcastle administrative records, directly from HCRA members, and website searches. RESULTS: In addition to advancing knowledge and providing capacity building support to members via grants, fellowships, scholarships, training, events and targeted translation support, key impacts of HCRA-member research teams included: (i) the establishment of a regional biobank that has distributed over 13,600 samples and became largely self-sustaining; (ii) conservatively leveraging $43.8 M (s.a.$20.5 M - $160.5 M) in funding and support from the initial $9.7 M investment; (iii) contributing to clinical practice guidelines and securing a patent for identification of stem cells for endometrial cell regeneration; (iv) shifting the treatment paradigm for all tumour types that rely on nerve cell innervation, (v) development and implementation of the world's first real-time patient treatment verification system (Watchdog); (vi) inventing the effective 'EAT' psychological intervention to improve nutrition and outcomes in people experiencing radiotherapy for head and neck cancer; (vi) developing effective interventions to reduce smoking rates among priority groups, currently being rolled out to disadvantaged populations in NSW; and (vii) establishing a Consumer Advisory Panel and Consumer Engagement Committee to increase consumer involvement in research. CONCLUSION: Using FAIT methodology, we have demonstrated the significant impact and downstream benefits that can be achieved by the provision of infrastructure-type funding to regional and rural research collaboratives to help address inequities in research activity and health outcomes and demonstrates a positive return on investment.

Improving Retention Through a Purposeful Multidisciplinary Mentorship Program.

In response to high nurse turnover, a 12-hospital health system team created a sustainable, formalized, and interprofessional mentoring model to improve nurse retention. In the 1st year of the program, 506 mentor-mentee matches were made, and nearly 5000 hours of mentoring time were logged. Data revealed that turnover was significantly reduced in both the mentor and mentee groups compared with employees who were not in the program.